Content delivery networks or content distribution networks (CDNs) deliver content (such as large media content) to end users. Nodes in a content delivery network cooperate with each other to satisfy requests for content by end users. Typically, content delivery networks employ one or more optimization techniques to improve the delivery process. For example, content delivery networks often optimize the delivery process to reduce the bandwidth costs, improve end-user performance, or both. When optimizing for end-user performance, for example, a user request is typically sent to a node that can deliver the content quickly to the user (e.g., by selecting a node that has the fewest hops or that is the fewest number of network seconds away from the requestor)
Nodes in a content delivery network often store content that has been requested by one user in a cache for subsequent delivery to one or more additional users. In this manner, other users served by the same node that want the same cached item can obtain the item from the cache. Existing caching schemes used by content distribution companies, such as Akamai Technologies, Inc., of Cambridge, Mass., are generally designed for relatively small content (on the order of mega bytes or less) and for Internet-type networks. Such content delivery networks provide effective caching, where requested content is often stored in clusters relatively close to end users. As a result, most requested files do not have to travel from the original server to reach each destination, saving in bandwidth usage at each hop and also reducing latency
As the popularity of next generation network applications increases, such as video on demand (VOD) and Internet television services, the number and size of the files to be stored in a cache will significantly increase. For example, a two hour full feature film in a reasonably high resolution format may be on the order of 5 GB (similar to the storage requirements of a DVD). Thus, caching capacity issues will become increasingly significant, with a caching capacity on the order of 100 s of Terabytes easily contemplated. Similarly, bandwidth savings due to good caching schemes can have a very large impact on bandwidth requirements in large-content delivery networks.
A need therefore exists for caching techniques in large-content delivery networks that effectively use the caches and reduce required bandwidth. Unlike small size content (KB to MB sizes), this type of content is typically owned by the video content owner/provider. Therefore, such content can only be cached in specific nodes within a typically proprietary network which is most likely architected for this kind of content delivery. To avoid the significant overhead of constant computation of best locations for specific items and their popularity distribution, as done today in content delivery networks, a need therefore exists for self-organizing caching techniques in such content delivery networks. In this manner, the data stored in the cache can be organized without significant monitoring or management.